445 research outputs found
Estimation of the Rate-Distortion Function
Motivated by questions in lossy data compression and by theoretical
considerations, we examine the problem of estimating the rate-distortion
function of an unknown (not necessarily discrete-valued) source from empirical
data. Our focus is the behavior of the so-called "plug-in" estimator, which is
simply the rate-distortion function of the empirical distribution of the
observed data. Sufficient conditions are given for its consistency, and
examples are provided to demonstrate that in certain cases it fails to converge
to the true rate-distortion function. The analysis of its performance is
complicated by the fact that the rate-distortion function is not continuous in
the source distribution; the underlying mathematical problem is closely related
to the classical problem of establishing the consistency of maximum likelihood
estimators. General consistency results are given for the plug-in estimator
applied to a broad class of sources, including all stationary and ergodic ones.
A more general class of estimation problems is also considered, arising in the
context of lossy data compression when the allowed class of coding
distributions is restricted; analogous results are developed for the plug-in
estimator in that case. Finally, consistency theorems are formulated for
modified (e.g., penalized) versions of the plug-in, and for estimating the
optimal reproduction distribution.Comment: 18 pages, no figures [v2: removed an example with an error; corrected
typos; a shortened version will appear in IEEE Trans. Inform. Theory
Recommended from our members
Entropy bounds on abelian groups and the ruzsa divergence
Over the past few years, a family of interesting new inequalities for the
entropies of sums and differences of random variables has been developed by
Ruzsa, Tao and others, motivated by analogous results in additive
combinatorics. The present work extends these earlier results to the case of
random variables taking values in or, more generally, in
arbitrary locally compact and Polish abelian groups. We isolate and study a key
quantity, the Ruzsa divergence between two probability distributions, and we
show that its properties can be used to extend the earlier inequalities to the
present general setting. The new results established include several variations
on the theme that the entropies of the sum and the difference of two
independent random variables severely constrain each other. Although the
setting is quite general, the result are already of interest (and new) for
random vectors in . In that special case, quantitative bounds are
provided for the stability of the equality conditions in the entropy power
inequality; a reverse entropy power inequality for log-concave random vectors
is proved; an information-theoretic analog of the Rogers-Shephard inequality
for convex bodies is established; and it is observed that some of these results
lead to new inequalities for the determinants of positive-definite matrices.
Moreover, by considering the multiplicative subgroups of the complex plane, one
obtains new inequalities for the differential entropies of products and ratios
of nonzero, complex-valued random variables
Recommended from our members
Approximating a diffusion by a finite-state hidden Markov model
© 2016 Elsevier B.V. For a wide class of continuous-time Markov processes evolving on an open, connected subset of Rd, the following are shown to be equivalent: (i) The process satisfies (a slightly weaker version of) the classical Donsker–Varadhan conditions;(ii) The transition semigroup of the process can be approximated by a finite-state hidden Markov model, in a strong sense in terms of an associated operator norm;(iii) The resolvent kernel of the process is ‘v-separable’, that is, it can be approximated arbitrarily well in operator norm by finite-rank kernels.Under any (hence all) of the above conditions, the Markov process is shown to have a purely discrete spectrum on a naturally associated weighted L∞space
Identifying statistical dependence in genomic sequences via mutual information estimates
Questions of understanding and quantifying the representation and amount of
information in organisms have become a central part of biological research, as
they potentially hold the key to fundamental advances. In this paper, we
demonstrate the use of information-theoretic tools for the task of identifying
segments of biomolecules (DNA or RNA) that are statistically correlated. We
develop a precise and reliable methodology, based on the notion of mutual
information, for finding and extracting statistical as well as structural
dependencies. A simple threshold function is defined, and its use in
quantifying the level of significance of dependencies between biological
segments is explored. These tools are used in two specific applications. First,
for the identification of correlations between different parts of the maize
zmSRp32 gene. There, we find significant dependencies between the 5'
untranslated region in zmSRp32 and its alternatively spliced exons. This
observation may indicate the presence of as-yet unknown alternative splicing
mechanisms or structural scaffolds. Second, using data from the FBI's Combined
DNA Index System (CODIS), we demonstrate that our approach is particularly well
suited for the problem of discovering short tandem repeats, an application of
importance in genetic profiling.Comment: Preliminary version. Final version in EURASIP Journal on
Bioinformatics and Systems Biology. See http://www.hindawi.com/journals/bsb
Geometric ergodicity in a weighted sobolev space
For a discrete-time Markov chain evolving on with
transition kernel , natural, general conditions are developed under which
the following are established:
1. The transition kernel has a purely discrete spectrum, when viewed as a
linear operator on a weighted Sobolev space of functions with
norm, where is a Lyapunov function and .
2. The Markov chain is geometrically ergodic in : There is a
unique invariant probability measure and constants and
such that, for each , any initial condition
, and all : where .
3. For any function there is a function solving Poisson's equation: Part of the
analysis is based on an operator-theoretic treatment of the sensitivity process
that appears in the theory of Lyapunov exponents
Compound poisson approximation via information functionals
An information-theoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Nonasymptotic bounds are derived for the distance between the distribution of a sum of independent integer-valued random variables and an appropriately chosen compound Poisson law. In the case where all summands have the same conditional distribution given that they are non-zero, a bound on the relative entropy distance between their sum and the compound Poisson distribution is derived, based on the data-processing property of relative entropy and earlier Poisson approximation results. When the summands have arbitrary distributions, corresponding bounds are derived in terms of the total variation distance. The main technical ingredient is the introduction of two "information functionals,'' and the analysis of their properties. These information functionals play a role analogous to that of the classical Fisher information in normal approximation. Detailed comparisons are made between the resulting inequalities and related bounds
Recommended from our members
Simulated convergence rates with application to an intractable α-stable inference problem
© 2017 IEEE. We report the results of a series of numerical studies examining the convergence rate for some approximate representations of α-stable distributions, which are a highly intractable class of distributions for inference purposes. Our proposed representation turns the intractable inference for an infinite-dimensional series of parameters into an (approximately) conditionally Gaussian representation, to which standard inference procedures such as Expectation-Maximization (EM), Markov chain Monte Carlo (MCMC) and Particle Filtering can be readily applied. While we have previously proved the asymptotic convergence of this representation, here we study the rate of this convergence for finite values of a truncation parameter, c. This allows the selection of appropriate truncations for different parameter configurations and for the accuracy required for the model. The convergence is examined directly in terms of cumulative distribution functions and densities, through the application of the Berry theorems and Parseval theorems. Our results indicate that the behaviour of our representations is significantly superior to that of representations that simply truncate the series with no Gaussian residual term
- …